Data Mining Project: Spaceship Titanic¶

Welcome to the year 2912, where our data science skills are needed to solve a cosmic mystery. We have received a transmission from four light-years away, and things aren't looking good. The Spaceship Titanic was an interstellar passenger liner launched a month ago. With almost 13,000 passengers on board, the vessel set out on its maiden voyage transporting emigrants from our solar system to three newly habitable exoplanets orbiting nearby stars. While heading towards its first destination, the scorching 55 Cancri E, the unsuspecting Spaceship Titanic collided with a spacetime anomaly hidden within a dust cloud. Unfortunately, it met a fate similar to its namesake from 1000 years before. Although the ship remained intact, almost half of the passengers were transported to an alternate dimension! With this challenge, we want to clusterize data in order to answers the question: “what sorts of people were more likely to survive?” using passenger data (ie name, age, gender, socio-economic class, etc). We want to analyze the data with a Python notebook to create clustering models and examine the received data. We will try to better understand the events that led to this cosmic situation and contribute to unraveling the mystery behind this space collision. The goal is to determine which passenger has been Transported to another dimension or not.

We are going to develope the report in 3 chapters:

1) In the first chapter we will focus on the exploration of the dataset, the distribution of the different features and their relations.

2) In the second one we will engage in feature engeneering, the filling of missing values and data preprocessing

3) In the third chapter we will perform clustering, and its validation with internal and external indexes. Interpretation of the final results.

Libraries¶

Chapter 1¶

Let's begin with the exploration of the datasets

The dataset is formed by 8693 entries, each one representing a passenger, and 14 different features. Some of which are categorical and some numerical. Their interpretation is the following:

• PassengerId - A unique Id for each passenger. Each Id takes the form gggg_pp where gggg indicates a group the passenger is travelling with and pp is their number within the group. People in a group are often family members, but not always.
• HomePlanet - The planet the passenger departed from, typically their planet of permanent residence.
• CryoSleep - Indicates whether the passenger elected to be put into suspended animation for the duration of the voyage. Passengers in cryosleep are confined to their cabins.
• Cabin - The cabin number where the passenger is staying. Takes the form deck/num/side, where side can be either P for Port or S for Starboard.
• Destination - The planet the passenger will be debarking to.
• Age - The age of the passenger.
• VIP - Whether the passenger has paid for special VIP service during the voyage.
• RoomService, FoodCourt, ShoppingMall, Spa, VRDeck - Amount the passenger has billed at each of the Spaceship Titanic's many luxury amenities.
• Name - The first and last names of the passenger.
• Transported - Whether the passenger was transported to another dimension. This is the target, the column you are trying to predict.

Since we want to perform unsupervised learning we do not want the label of the target, so we drop it. The coulmn 'Name' being a categorical feature specific for every entry could hardly be useful in any way during the process, so we drop it as well.

As a first step, for a better understanding, we divide the features in numerical and categorical.

Here we create 2 lists with the name of categorical features and numerical features

Numerical features¶

Let's now plot the distribution for every numerical feature.

Age¶

RoomService¶

FoodCourt¶

ShoppingMall¶

Spa¶

VRDeck¶

As we can clearly see, the 5 "services" features have the same distribution. With a small amount of people who spent a lot and the major part spending zero. Suspecting that the this five features could be related and that the same people who spend in a field also spend in others, we would be tempted to construct a single "expenses" feature. Let's the analyze the combined ditribution and the correlation between them.

From the plot of the combind distibution between couples of variables we notice the lack of correlation. This is indeed confirmed by the correlation matrix which shows that the highest value is around 0.35. We conclude that the construction of a single feature would loose information and the expenses feature must be kept separate.

Categorical Features¶

We know focus on the catgorical features, starting with the ones that do not requre special attention for our purposes.

HomePlanet¶

Earth is the most common planet to depart from.

CryoSleep¶

Destination¶

TRAPPIST-1e is by far the most common destination.

VIP¶

We see that only a small percentage of passenger have a VIP card.

Let's now focus on the two most elaborate categorical features. Passenger ID and Cabin.

Passenger ID¶

We know that Passenger ID is composed by 6 digits, the first 4 labeling the number of the group and the last two indicate the number of a the person inside is group. So we create two different features respectively, 'Group' and 'Member', and plot their distribution.

From theese plots we see that there are ~9000 groups and most of them are composed by a single member. The maximum number of members in a group is 8 and the member distribution follows a negative exponential.

We now drop 'Passenger ID' and add the new two features.

Cabin¶

Folowing the same logic, from 'Cabin' we build three different features: 'Deck' (deck of the cabin), 'Num' (number of cabin in the deck), 'Side' (which can be 'S' pr 'P').

Now let's plot produce some scatter plots to see the relation between the new features.
We will use stripplot from seaborn library in order to avoid overlapping, and understand the density of the points.

From this plot we notice that the the decks are not evenly populated. The most poplated is the 'F' while the 'T' has only a handful of passengers. As a consequence ther are more cabins with a low number since they are present in every deck while high numbers can be reached only in the more populated decks. The side is instead quite evenly distributed.

We now merge the 3 new features to the dataset.

Chapter 2¶

Having explored the dataset we will now concentrate on the preprocessing of data.
We start by analyzing the missing values present in our dataset and finding the smartest way possible to deal with them. We will better analyze all the relations between the features and also we will take into account the relations between the new build features and the others.

Missing Values¶

We see that all the features have missing values apart from 'Group' and 'Member' which in our original datasets were part of 'Passenger ID'. Now let's count the missing values for every column.

Now we will try to count how many missing values are present in the same rows, in order to decide if we can drop the entire row or if we have to fill the missing values.

The missing values are around 200 for every feature, but they are not concentrated on the same entries. Infact, as we can see from the histogram most of the passengers with missing values have only one while passengers with 2, 3, 4 and 5 are less and less probable. As a consequence around 2000 entries have at least a missing value and eliminating every one of them will make us loose a significant part of the dataset. At the same time a passenger has 13 features. Since we saw that, excluding the very few passengers with 4 and 5 missing values, the maximum number of missing feature for a single passanger is 3, the percentage of missing feature is relatively low. So a good way to deal with the problem seems to be eliminating the few passengers with 4 and 5 missing features and try to fill all the other missing values in the most efficient way.

Fill missing values¶

Let's check if the new columns we introduced by separating 'Passengerer ID' and 'Cabin' brought to light correlations between features.

We indeed notice a strong correlation between 'Group' and 'Num'. Let's try to visualize it.

Group and Cabin Number¶

We clearly see that as we increase the group number, the cabin number increases too, but with different coefficients. From the plot of the cabin number per deck we saw how, for each deck, the cabin number went from 0 to a certain value and decks were differently populated and so had different maximum cabin number. We then strongly suspect that the different slopes here represent the different decks. Let's then plot the points with different colors depending on the deck.

This is a remarkable plot that confirms our thoughts. We then foresee a strategy to fill ther missing cabin values: when we have an entry with a missing cabin values(so he misses 'Deck', 'Num', 'Group' ) we check his group number. We know that passengers in the same group are not necessarely put in the same cabin but are for sure close, so most certainly in the same deck and side. So we explore other members of his group and take the most common deck among them to fill 'Deck' and the most common side to fill 'Side'. Then once we have the 'Deck' we have the specific slope of the line that relates 'Group' and 'Num', so we can fill 'Num' with the closest integer to y=mx + b where x is 'Group' and m and b are derived from the specific deck. If the passenger is alone in the group we just fill the deck with 'F' wich is by far the most common and randomly choose the side between 'S' and 'P' since they are evenly distributed. Then we apply the same logic to fill 'Num'.

For this process it is useful to introduce a feature that tells us directly if a member is alone in a group or not.

Let's now store the different lines for the different decks and perform a linear fit to obtain the slope and intercept parameters for each one of them. We then store the parameters in a dictionary

We see that there is only 5 passenger in deck 'T' which makes the line totally unprecise, but as we see in the cell below the problem of how to fill the 'Num' of passengers with the deck in T does not even come up, since there is no passenger with a missing cabin value and which is in a group in which the majority of other passengers have 'T' as deck.
In the for loop below we added a variable that counts the 'T' deck values added and it is indeed 0.

Since 'Group' has no missing values we filled all the cabin missing slots.